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ABSTRACT 

The plan calls for a six stage development process. 
Stages one and two represent a needs assessment for testing^ in which 
the target population and general subject areas to be assessed are 
determined. The remaining four stages are performed for each test 
developed. They include: item pool development r item filtering at the 
technical and lay levels ^ and actual test performance by item. The 
plan for test development is based on two assumptions: (1) that with 
all the tests that have been developed over the years, there are many 
good or modifiably good items available in most assessment areas; and 
(2) it is easier for people to f:elect items relevant to a curriculum 
and culture than it is for them to write such items. In order to 
assess the testing needs in the Department of Education, leading 
members of the administration were interviewed and asked to rate the 
importance of assessing the various subject matter areas. In 
addition, they were asked to indicate at what grade levels such 
assessment should take place. Based on this information and item 
availability, a minimal test construction program was adopted for the 
^9^2-^3 school year to test the model. It consisted of creating 
assessment instruments in English reading and science at the sixth 
and ninth grade levels. This program was begun in October 1972. A 
similar development program is planned in. other areas over the next 
two years. (Author/JM) 
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^ BACKGROUND 

Ll/ American Samoa is an unorganized, .unincorporated United 

States territory which lies 2300 miles sputh of Hawaii and 
1500 miles north of. New Zealand. It consists of seven 
islands of 76;2 square miles and now has a popula t:" -i of 
more than 27,159 of whom about 90% are Samoan by he^rifcc^ge 
and culture (Census, 1972). English is a second language 
for most Samoans, being used in schools and for official 
business while Samoan is the language of the home and of 
common usage . 

« 

There are presently 27 public elementarry schools with 
a total enrollment of nearly 6,000 students and four oublic ^ 
-high schools which serve about 2,200 students (Atisano'e, 
1972). Centrally devised and planned T.V- lessons are the 
• main source of instruction at the elementary school level, 
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while in high schools teachers plan their own work within 
general course outlines. By stateside standards, teacher 
preparation and training are still inadequate. Most 
elementary teachers have only a high-school diploma and a 
year of teacher training. English language skills are weak 
as evidence by a median reading grade equivalent score of 5.3 
on the SRA Achievements tests in the Spring of 1972. This is 
important because English is the only medium of instruction 
from grade 3 on. The high schools still employ many contract 
stateside personnel or their wives, most of whom were'hired 
specificly for the classes they are now teaching. 

While many problems remain to be solved, the system is 
now able to provide a steadily improving quality of universal 
education. Incentives have beer, i-n.troduced aa?ic5 .:i-nda.xds 
es l-.ablisVod to ge' » -is/;t.--r£ ••-o .u^c.i aii-e ifcrieicr si.SiLH-, ^-.a ,an A . 
or B.A. ->,_.vel withix "he next f esi y -^^r s . Sanite,:ans =e,xe ceplac— ig 
contract personnel in many areas and now hold over 85% of all 
positions, including all ^op administrative positions within 
the department (DOE, 1972). All elementary classroom teachers 
and principals are Samoan except for those teaching in 
Fia Ilea, a dependents school. A bilingual project is being 
designed to increase the teaching of Samoan language and 
understanding of Samoan culture in the schools. 



THE PROBLEM 

The Department of Education is faced with designing 
an assessment program, for a relatively small number of 
students, which if it is to be effective must consider 
Samoan language and culture, establish local norms, and 
be relevant to stateside , norms , as well as consider such 
factors as distance from outside resources, lack of local, 
staff with training, and financial restrictions. The 
department needs a program which will gather information 
about student progress and curriculum effectiveness so that 
the territory's resources can be better allocated and so 
the quality uf education the department provides can be 
upgraded. 

The Department of Education also needs data t.n 
establish its own standards or norms for its programs. 
Stateside norms do have some relevance for high school 
seniors, since there is a large stateside migration. Many 
of these who move are high school graduates who go to 
Hawaii or California to live (Wolf, 1969). But local norms 
are needed for Rnglish-as-a-Second-Language (ESL) students 
w'.iich take into account the local curriculum and the culture 
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FREVlOnS SOLUTIONS 

The standardized tests that have been given over the 
last few years have proved to be of little informational 
value for measuring system progress or curriculum effective- 
ness^ They tend to be too difficult for Samoan ESL 
students in all a.reas except Mathematics. In other subject 
matter tests, scores were low and correlated highly with 
EnglisJi reading results, probably reflecting the students* 
lack of stateside language proficiency. Many test items 
did not represent curricul uin o u t c o itie 3 alone, but reflected 
learning - cu], tural , iria tura tional , and factual - exce'p*:ed 
to occur outside of the school- A large proportion of the 
test materials were not covered .in the loc:a.i cjar.r±cu luiri . 
A ..Ttudv of the reF.ul-rs m-sxely coT::fix:ined v;;a:S alxeaxry known, 
namely that such te<>ts were generally inappropia te , at the 
elementary school level. Their effect was to downgrade the 
efforts and progress that had been made in the system and 
to introduce a counter-cultural and competitive situation 
which stressed attaining good scores, (Baldauf, & Reupena, 
1972) , - 

For the past several years^ locally made, system vide, 
tests were constructed by instructional television (ITV) 
personnel to measure course performance at a given grade 
level. These tests were course specific, providing no 
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generalizabili ty , and were often psychome tr ically unsound 
(Baldauf & Reupena? 1973). 

■^either of these programs was designed to specifically 
meet the needs of the department. The problem still remains 
to create the necessary assessjnent instruments given limited 
funding and staff resources. * 

PLANNING^ 

During I97J— 1972, the Education Revsfearch and Develop- ' 
ment Center (ED-RAD) at the University of Hawaii was con- 
tracted to hel]^ the DOE develop an as ^^:n ii -s 3sren t px-o g r:£rT: l:o 
meet its parti cuL^ir ne-eds . After e>x.a.m±n:rjni g possible:, 
altern ti^-:.3, ccrns.ulti:!..ng vrith Nstio:rEa£ii Assessment personnel, 
and surveying expr'}:}ssed needs, a plan was developed for 
creating norm referenced assessment instruments for 
American Samoa. 

The plan presented to the DOE in January, 1972 by 
Dr. Peter Dunn-Rankin of EDRAD was modified and adopted, 
A summary of it is given in Figure 1. It calls for a six 
stage development process. Stages one and two represent a 
needs assessment for testing, in which the target population 
and general subject areas .to be assessed are determined. 

2 ' ' ~ 

Materials in the following sections are adapted from the 
Section of the EDRAD report (1972) written by Dr. Peter 
Dunn-Rankin." (p. 44-53). 
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Oecide on Target Population: (Grade Levels 
6j'3?-9t12 suggested) . • 

t 

Determine subjects assessed: '(English, Saihoan 
Language .Arts, Mathematics, .Soc ial Studies , and 
Science suggested) . 

Collect Item Pool from variety of Sources. 



'urr icul am 
Tests 



P a c . Is. 
Ach. Tests 



^Jational 
As se ssme^it 
Tests 



Item Pool by Siubject 
and 

grade Levels - 1(1-4)^ i:i ( 3- 7 ) — l(6-li);; 




Filter olttem'S through 



DOE Objectives 



(1) DOE objectives, (2) 
needs assessment 
survey and (3) item 
writers . 



Needs Assessment 
Technical Review Committee 



Measurement Analysis 
Check on. content, context, difficulty level 
discrimination 



5. Let representatives of lay public, teachers, and 
students choose from subset of items ► 



Representatives of Publ^ic, 
Teachers, Students ^ 



j: 



Assess me nt Items 



6. Use a set of final items as an assessment measure 
Report performance by item. 
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Figure 1-- Rough Plan for Curriculum Assessment in 
American Samoa. (From EDRAD, 1972) 



The remaining four stages are performed for each test 
developed. They include: item pool development, item 
filtering at the technical and lay levels and actual test 
performance by item. 

RATIONALE^ 

The plan foo*- test ..dev.e^.opim'- nt ou~'iin:ed above is the 
opposite of the usual strategy for test construction. 
Items are usually written to measure specific curriculum 
objectives. This takes considerable resources in terms 
of time, money ar>d personnel if it is to be done properly 

The strategy adopted makes two basic assumptions, 

(a) That with all the tests that have. been 

developed over the years, there are many 
good or modifiably good items available 
in most assessment areas. 
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(b) It is easier for people to select items relevant 
to a curriculum and culture than it is for them 
to write such items. 

This strategy satisfies the needs of the Samoan assess- 
ment program, in that it pro-ides a method to obtain 
culturally relevant measures of the school curriculum taught 
in American Samoa, it furnishes local norms or standards, 
and it is a program which the Department of Education can 
afford. 

SUBJECT AREAS AND GRADE LEVELS 
In order to assess the testing needs in the Department 
of Education, leading members of the administration were 
interviewed and asked to rate the importance of assessing 
the various subject matter areas. In addition, they were 
asked to indicate at what grade levels such assessment 
should take place. The results of these ratings are pre- 
sented in Figure 2. (EDRAD, p. 47). 

From this scale it is readily apparent that the languae 
arts areas, both in English and Samoan, are felt to be the 
most important to assess, while subject matter areas are of 
secondary imortance. Raters also indicated that the levels 
assessed should be 6th grade, 3rd grade, 8th or 
9th grade and in that order of importance . This reflects 
the feeling that assessment should come early enough so 
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Reading English 



Speaking English; Writing English 



Reading Samoan 
Writing Samoan 

Career Development ; Speaking Samoan 
Math 

Social Studies 
Science 

Health & Physical Education; Consumer 
Education; Vocation Arts 

Literature 
Art; Music 



Figure 2--Relative Importance of Curriculum Areas for 

Assessment in. American Samoa, Order of 

Grade LeVel Importance is 6*'3>9>8>12, 
(from EDRAD, 1972) 
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that difficulites can be ascertained and corrected. (EDRAD, 
p. 49), 

Based on this information and item availability, a 
miminal test construction program was adopted for the 
1972-1973 school year to test the model. It consisted of 
creating assessment instruments in English reading and 
science at the 6th-and 9th- grade levels,. This program was 
begun in October 1972. The materials which follow describe 
what has occurred and what is planned for the rest of the 
year in one specific area of the development program, 
English reading assessment ' for grades 6 and 9. A similar 
development program is planned in other areas over the next 
two years. A summary of the tests' to be developed if funding 
is avilable is given in Appendix A. . 

CREATING THE ITEM POOL*^ 

In November and December of 1972 the initial item pool 
was created. Items were gathBred from a number of sources, 
including standardized reading tests. National Assessment 
of Educational Progress materials (NAEP,1972), Samoan 
curriculum tests, and other tests obtained from Pacific 
Island Depc^rtments of Education. 



Work in this area was performed by EDRAD, University of 
Hawaii. 
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Items were then judged according to their difficulty 
level. Items with a difficulty level below 4th grade or 
above the 9th grade were eliminated from the initial item 
pool as probably inappropiate tor the tests being developed 
This resulted in a group of .1453 items. These items were 
then grouped by item type, using the eight NAEP themes 

ITEM SELECTION - STAGE I 

The first stage of the item selection process was to 
have "experts" rate all 1453 items. A form was designed 
for this purpose which asked the raters to place each item 
in one of three categories: "acceptable" (High or Low), 
"acceptable" if modified" (High or 'Low) or "reject". Raters 
were asked to use the following criteria in arriving at an 
assessment of each it-m: (a) relevance to curriculum, 
(b) . relevance to culture, and (c) technical quality of the" 
i tern. 

Two committees of six members each were set up, one at 
the University of Hawaii, composed of specialists in reading 
and measuremeiit and one in American Samoa composed of 
trained teacher:- and experienced teacher-administrators. 
Samoans and non-Samoans were included on each committee. 

Each of the persons spent about -8 to 10 hours rating 
all the items. The ratings for each i,tem will then be 
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summarized. The plan calls f rom furth.er 

consideration items rejected ^ the raters. The 

remaining items will then be. given weighted scores based on 
the number of high and low "acceptable" or "acceptable if 
modified" tallies they received. Items with low scores will 
be rejected leaving a revised item pool of about 200 to 300 
items for each grade level. Items with "acceptable if 
modified" ratings will be altered to make them more accept- 
able; 

ITEM SELECTION - STAGE II 

This part of the development process is scheduled for 
late April, 1973. The items from the revised item pools 
will be grouped in booklets of 50 items each. These book- 
lets will be given to informed groups' of Samoan lay persons 
who will be asked to rate the items in each booklet on the 
basis of whether they feel each item measures something 
the student should be learning in school. 

These ratings will be summed for allpersons rating 
each of the items to get a measure of the items accept- 
ability for the community. 
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TEST WRITING 

The; evaluation committee in American Samoa, with the 
aid of an outside measurement expert, will be responsible 
for selecting the final 50 to 75 items to be used in each 
test. Items will be selected based on Samoan lay group 
^atJ.ngs,-^^d balanced to reflect curriculjum emphases. Items 
will be reviewed and modified where necessary to reflect 
■ proper vocabulary level, Samoan culture, and technical item 
construction cons ider ations - 

TEST ADMINISTRATION AND ANALYSIS 

The tests will be administered to a stratified random 
sample of 6th- and 9th- grade students in schools selected 
from geopraphical groupings which cover the range of 
traditional to more western • cultural life styles currently 
followed in Samoa. ^ 

The results will.be analyzed in a manner similar to 
that followed by the National Assessment program. Results 
will stress item analysis and the relationship of items 
to the curriculum. The basic data then will not be student 
scores or school means, but norms based on student abilities 
to handle each aspect of a curriculum area. These data will 
allow the department of education to assess its curriculum 
^j^^-goals and provide information on which constructive 
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changes can be made. They will also provide standards or 
norms by which the der ment may judge groups of students 
in years to come. 



CONCLUSION 



The test development plan outlined above has grown 
out three convictions. First some type of measurement is 
necessary for educational programs if there is to be 
planned progress in providing better education for our 
students. Second, the assessment instruments previously 
used -were not providing the information needed and were 
perhaps having a negative effect on the educational system. 
Third, the competitive situation which was created between 
schools by standardized test programs goes directly against 
the Samoan cultural belief in co-operation and in hot 
making any person or group seem less worthwhile then any 
other. 

It is hoped that the ideas in this paper will provide 
a stimulus to the development of better evaluation 
techniques and test materials in all our school sytem. 
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'APPENDIX. A 

Proposed v:- iterion Assessment Plan for American Samoa 
Years 72-73 73-74 74-75 



Grades 



k 
1 

2 



Reading Sam . 
Reading Eng , 



Math 



Reading Sam. 
Speaking Sam, 
Reading Eng . 
Speaking Eng , 
Math 



Reading Eng 



Science 



7 
8 



PN^eading Eng . 
Reading Sara, 
Speaking Eng 
Science, 
Math 



Reading Eng . 
P^neaking Eng. 
Writing Eng. 
Reading Sam. 
Speaking Sam • 
Science 
Math-Soc. St. 



Reading Eng , 



Science 



10 
11 



12 



Reading Eng . 
Speaking Eng 

Science 
Math 



Reading Eng . 
Career Dev. 
Science 
Math 
Soc. St, 



Reading Eng . 
Speaking Eng , 

Science 
Math 
Soc. St. 



Reading Eng 
Writing Eng. 
Career Dev. 
Science 
Math 

Soc. St. 



ERJC From' EDRAD, 19 72; 



pg. 140) 



